{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "596f9a2a",
   "metadata": {},
   "source": [
    "(automate-project-git-source)=\n",
    "# Create a project using a Git source\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a19681b6",
   "metadata": {},
   "source": [
    "This example shows how to create a project in MLRun for CI/CD automation based on a remote source, in this case Git.\n",
    "The process is equivalent to using tar.gz, zip archive files. This example assumes you have functions that are ready \n",
    "to run, and that you already cloned the files to your local file system.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694b3829",
   "metadata": {},
   "source": [
    "The flow covers these steps to create a CI/CD project in MLRun:\n",
    "1. [Before you start](#before-you-start)\n",
    "2. [Creating a project](#creating-a-project)\n",
    "2. [Setting a project source: remote or archive](#setting-a-project-source-either-remote-or-archive)\n",
    "3. [Setting and running functions using Git source code](#setting-and-running-functions-using-git-source-code)\n",
    "4. [Setting a workflow](#setting-a-workflow)\n",
    "4. [Running a workflow using a Git source](#running-a-workflow-using-a-git-source)\n",
    "5. [Setting and registering the project artifacts](#setting-and-registering-the-project-artifacts)\n",
    "6. [Creating and saving the project YAML](#creating-and-saving-the-project-yaml)\n",
    "7. [Creating and pushing changes to your Git repo or archive file](#creating-and-pushing-changes-to-your-git-repo-or-archive-file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d0d6dc",
   "metadata": {},
   "source": [
    "## Before you start"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "461c29a2",
   "metadata": {},
   "source": [
    "Install MLRun. If MLRun is not installed use ``pip install mlrun==<mlrun server version>`` or <br>``sh align_mlrun.sh`` \n",
    "(the default MLRun installer that automatically installs the server version).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cc9586ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlrun"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0317e09",
   "metadata": {},
   "source": [
    "Before running this notebook, clone the Git repo to your local machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c7eb4967",
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete the clone folder if exists\n",
    "!rm -rf ./clone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "61b6eea1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into './clone'...\n",
      "remote: Enumerating objects: 209, done.\u001b[K\n",
      "remote: Counting objects: 100% (209/209), done.\u001b[K\n",
      "remote: Compressing objects: 100% (150/150), done.\u001b[K\n",
      "remote: Total 209 (delta 118), reused 129 (delta 53), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (209/209), 162.20 KiB | 1.65 MiB/s, done.\n",
      "Resolving deltas: 100% (118/118), done.\n"
     ]
    }
   ],
   "source": [
    "# clone the repo to your local machine\n",
    "!git clone https://github.com/mlrun/ci-cd-tutorial.git ./clone"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "609a1704",
   "metadata": {},
   "source": [
    "## Creating a project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "26f79d2e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:04:46,772 [info] loaded project new-ci-cd-proj from MLRun DB\n"
     ]
    }
   ],
   "source": [
    "# Create a new project or load it from DB\n",
    "project = mlrun.get_or_create_project(name=\"new-ci-cd-proj\", context=\"./clone\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a6570a",
   "metadata": {},
   "source": [
    "or for loading from a private repo:\n",
    "\n",
    "```\n",
    "# project = mlrun.get_or_create_project(name='new-ci-cd-proj',context='./',init_git=True,secrets={\"GIT_TOKEN\":<github-token>})\n",
    "```\n",
    "\n",
    "See more details in {py:class}`~mlrun.projects.get_or_create_project` and {ref}`secrets`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a8c8c00",
   "metadata": {},
   "source": [
    "When you create a new project, MLRun creates a light project YAML, for example: \n",
    "````\n",
    "kind: project\n",
    "metadata:\n",
    "  name: new-ci-cd-proj\n",
    "  created: '2022-06-30T09:41:05.612000'\n",
    "spec:\n",
    "  functions: []\n",
    "  workflows: []\n",
    "  artifacts: []\n",
    "  desired_state: online\n",
    "status:\n",
    "  state: online\n",
    "````\n",
    "\n",
    "As you proceed, more information (project metadata) is added to the project YAML."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b10081bc",
   "metadata": {},
   "source": [
    "## Setting a project source, either remote or archive "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5697340c",
   "metadata": {},
   "source": [
    "Define the project source, and optionally `pull_at_runtime` flag value, and the project working dir, and add them to \n",
    "the `project.yaml` by using the {py:class}`~mlrun.projects.MlrunProject.set_source` method.   \n",
    "To copy these values also to the functions spec, set `with_repo=True` in the `project.set_function` method. \n",
    "\n",
    "- If `pull_at_runtime=True` MLRun loads the git/archive repo into the function container at run time and does not require a build. (This is simpler when developing, for production it’s preferable to build the image with the code.)\n",
    "- If `pull_at_runtime` is **not set to `True`**, you need to deploy the functions (with `function.deploy()`) to build a container.\n",
    "\n",
    "See more about `pull_at_runtime` in [Loading the code from container vs. loading the code at runtime](./git-best-practices.html#loading-the-code-from-container-vs-loading-the-code-at-runtime).<br>\n",
    "See also {py:class}`KubejobRuntime.with_source_archive <mlrun.runtimes.KubejobRuntime.with_source_archive>` and {py:class}`RemoteRuntime.with_source_archive <mlrun.runtimes.RemoteRuntime.with_source_archive>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ac7f7aed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add the git branch or references to the source e.g.: 'git://url/org/repo.git#<branch-name or refs/heads/.. or refs/tags/..>`.\n",
    "source = \"git://github.com/mlrun/ci-cd-tutorial.git\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d735fcac",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the project source\n",
    "project.set_source(source=source, pull_at_runtime=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ac2414b",
   "metadata": {},
   "source": [
    "## Setting and running functions using Git source code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "551d1467",
   "metadata": {},
   "source": [
    "This section describes: [fetching the data](#fetching-the-functions-data); [running the function](#running-the-function); [training the model using the fetched data](#training-the-model); and [serving the function](#serving-the-function).\n",
    "\n",
    "The code source files are usually stored under a folder named `./src` in the project context, <br>\n",
    "for example: `./project-context/src/data_fetch.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43e698b8",
   "metadata": {},
   "source": [
    "### Setting the project's functions\n",
    "\n",
    "To set the function's definitions, use the {py:class}`~mlrun.projects.MlrunProject.set_function` method. \n",
    "This sets the function's metadata in the project YAML, for example: \n",
    "function source (YAML, py, ipynb, function object), name of the function, function handler, function image, \n",
    "function kind, and function requirements.\n",
    "\n",
    "See more details in {py:class}`~mlrun.projects.MlrunProject.set_function`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "671d5f4e",
   "metadata": {},
   "source": [
    "### Fetching the function's data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a2c81b7",
   "metadata": {},
   "source": [
    "```{admonition} Tip: Using tags \n",
    "This example includes a `tag` value that is used as the Git tag for the release after completing the development. The tag \n",
    "must be added manually to the function. (This tag is internal to MLRun and is not taken from Git.)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9769cdd7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f08d7a1ae50>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set data_fetch function to the project.yaml file\n",
    "project.set_function(\n",
    "    func=\"./src/data_fetch.py\",\n",
    "    name=\"data-fetch\",\n",
    "    handler=\"data_fetch\",\n",
    "    kind=\"job\",\n",
    "    image=\"mlrun/mlrun\",\n",
    "    with_repo=True,\n",
    "    tag=\"v4\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e74fad55",
   "metadata": {},
   "source": [
    "### Running the function\n",
    "\n",
    "After you set the function in the project, get the function object with the \n",
    "{py:class}`~mlrun.projects.MlrunProject.get_function` method.\n",
    "\n",
    "\n",
    "```{admonition} Tip: Change the function spec with get_function\n",
    "You can use the `get_function` method to change the function spec. For example, if you  \n",
    "change the function resources and then run the function, it runs with those changes and the changes are stored in \n",
    "the project object cache:\n",
    "\n",
    "      \n",
    "      data_fetch_func = mlrun.get_function('data-fetch')\n",
    "      data_fetch_func.with_requests(mem='1G',cpu=3)\n",
    "      data_fetch_run = project.run_function('data-fetch')\n",
    "      \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68b61b18",
   "metadata": {},
   "source": [
    "Run the function using the {py:class}`~mlrun.projects.MlrunProject.run_function` method both to \n",
    "[run jobs locally](#running-the-function-locally) \n",
    "and [remotely on the runtime/cluster](#running-the-function-remotely-on-your-cluster). If \n",
    "there are any requirements, you need to build a new \n",
    "image before you run a function. See more details in {ref}`build-function-image`.\n",
    "\n",
    "#### Running the function locally\n",
    "\n",
    "First, run the function using the code files from your project context folder on your local file system, for debugging the function. Once you are satisfied, continue with [Running the function remotely on your cluster](#running-the-function-remotely-on-your-cluster). \n",
    "\n",
    "To run the code locally, use `local=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "baa498e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:04:46,993 [info] Storing function: {'name': 'data-fetch-data-fetch', 'uid': 'a6e5cc8f573e41f6ae6ef1c049b6e50a', 'db': 'http://mlrun-api:8080'}\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>new-ci-cd-proj-shapira</td>\n",
       "      <td><div title=\"a6e5cc8f573e41f6ae6ef1c049b6e50a\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/a6e5cc8f573e41f6ae6ef1c049b6e50a/overview\" target=\"_blank\" >...49b6e50a</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>May 17 09:04:47</td>\n",
       "      <td>completed</td>\n",
       "      <td>data-fetch-data-fetch</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=shapira</div><div class=\"dictlist\">kind=</div><div class=\"dictlist\">owner=shapira</div><div class=\"dictlist\">host=jupyter-shapira-7fc985f9db-cp8x9</div><div class=\"dictlist\">release=v3</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div title=\"v3io:///projects/new-ci-cd-proj-shapira/artifacts/data-fetch-data-fetch/0/train-dataset.parquet\">train-dataset</div><div title=\"v3io:///projects/new-ci-cd-proj-shapira/artifacts/data-fetch-data-fetch/0/test-dataset.parquet\">test-dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultd95d80d6-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultd95d80d6-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultd95d80d6\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultd95d80d6-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/a6e5cc8f573e41f6ae6ef1c049b6e50a/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:04:50,355 [info] run executed, status=completed: {'name': 'data-fetch-data-fetch'}\n"
     ]
    }
   ],
   "source": [
    "data_fetch_run = project.run_function(\n",
    "    function=\"data-fetch\", returns=[\"train-dataset\", \"test-dataset\"], local=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5953eab",
   "metadata": {},
   "source": [
    "#### Running the function remotely on your cluster\n",
    "\n",
    "```{admonition} Tip: Using a relative handler\n",
    "If your code is saved to your remote source, you can run the function from a remote source by adding \n",
    "`with_repo=True`. You can also specify a relative handler (folder_name.file_name.function_handler) to point to the python \n",
    "code file. (This paradigm does not support running functions in local.)\n",
    "\n",
    "    \n",
    "      project.set_function(name=\\\"training\\\",\n",
    "          handler=\\\"function.model_training\\\",\n",
    "          image=\\\"mlrun/mlrun\\\", kind=\\\"job\\\",with_repo=True\n",
    "          )\n",
    "      \n",
    "```\n",
    "Use the code files from the remote project source (`local=False`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "28a916b8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:04:50,395 [info] Storing function: {'name': 'data-fetch-data-fetch', 'uid': '860b9700cd3f4724a0669cb7c9732beb', 'db': 'http://mlrun-api:8080'}\n",
      "> 2023-05-17 09:04:50,649 [info] Job is running in the background, pod: data-fetch-data-fetch-qd874\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>new-ci-cd-proj-shapira</td>\n",
       "      <td><div title=\"860b9700cd3f4724a0669cb7c9732beb\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/860b9700cd3f4724a0669cb7c9732beb/overview\" target=\"_blank\" >...c9732beb</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>May 17 09:04:54</td>\n",
       "      <td>completed</td>\n",
       "      <td>data-fetch-data-fetch</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=shapira</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=shapira</div><div class=\"dictlist\">mlrun/client_version=1.3.1-rc5</div><div class=\"dictlist\">mlrun/client_python_version=3.7.6</div><div class=\"dictlist\">host=data-fetch-data-fetch-qd874</div><div class=\"dictlist\">release=v3</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div title=\"v3io:///projects/new-ci-cd-proj-shapira/artifacts/data-fetch-data-fetch/0/train-dataset.parquet\">train-dataset</div><div title=\"v3io:///projects/new-ci-cd-proj-shapira/artifacts/data-fetch-data-fetch/0/test-dataset.parquet\">test-dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result92a33882-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result92a33882-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result92a33882\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result92a33882-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/860b9700cd3f4724a0669cb7c9732beb/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:05:03,834 [info] run executed, status=completed: {'name': 'data-fetch-data-fetch'}\n"
     ]
    }
   ],
   "source": [
    "data_fetch_run = project.run_function(\n",
    "    function=\"data-fetch\", returns=[\"train-dataset\", \"test-dataset\"], local=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a7ccfb7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'train-dataset': 'store://artifacts/new-ci-cd-proj-shapira/data-fetch-data-fetch_train-dataset:860b9700cd3f4724a0669cb7c9732beb',\n",
       " 'test-dataset': 'store://artifacts/new-ci-cd-proj-shapira/data-fetch-data-fetch_test-dataset:860b9700cd3f4724a0669cb7c9732beb'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_fetch_run.outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "75b31f43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.655400</td>\n",
       "      <td>1.357176</td>\n",
       "      <td>-0.380252</td>\n",
       "      <td>2.236612</td>\n",
       "      <td>0.102893</td>\n",
       "      <td>-0.038678</td>\n",
       "      <td>0.101061</td>\n",
       "      <td>1.555770</td>\n",
       "      <td>1.116734</td>\n",
       "      <td>0.146883</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.132336</td>\n",
       "      <td>0.739735</td>\n",
       "      <td>0.452615</td>\n",
       "      <td>0.299427</td>\n",
       "      <td>0.683967</td>\n",
       "      <td>-0.089078</td>\n",
       "      <td>0.609046</td>\n",
       "      <td>-0.895865</td>\n",
       "      <td>-0.578405</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.834107</td>\n",
       "      <td>0.572216</td>\n",
       "      <td>-0.872751</td>\n",
       "      <td>0.519342</td>\n",
       "      <td>-1.101798</td>\n",
       "      <td>0.259935</td>\n",
       "      <td>0.398852</td>\n",
       "      <td>-0.299485</td>\n",
       "      <td>0.821154</td>\n",
       "      <td>0.018271</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.634293</td>\n",
       "      <td>-0.618584</td>\n",
       "      <td>1.354337</td>\n",
       "      <td>-1.136238</td>\n",
       "      <td>1.248243</td>\n",
       "      <td>-0.593805</td>\n",
       "      <td>0.266741</td>\n",
       "      <td>1.180665</td>\n",
       "      <td>1.212383</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>-2.040370</td>\n",
       "      <td>3.446461</td>\n",
       "      <td>-0.269668</td>\n",
       "      <td>-0.875862</td>\n",
       "      <td>1.347329</td>\n",
       "      <td>1.412033</td>\n",
       "      <td>0.764714</td>\n",
       "      <td>2.161531</td>\n",
       "      <td>0.390874</td>\n",
       "      <td>-0.900138</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.904111</td>\n",
       "      <td>2.640541</td>\n",
       "      <td>-2.483898</td>\n",
       "      <td>-1.619484</td>\n",
       "      <td>-3.676358</td>\n",
       "      <td>0.704040</td>\n",
       "      <td>-3.192003</td>\n",
       "      <td>1.669527</td>\n",
       "      <td>0.782062</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           0         1         2         3         4         5         6  \\\n",
       "14  0.655400  1.357176 -0.380252  2.236612  0.102893 -0.038678  0.101061   \n",
       "27  0.834107  0.572216 -0.872751  0.519342 -1.101798  0.259935  0.398852   \n",
       "79 -2.040370  3.446461 -0.269668 -0.875862  1.347329  1.412033  0.764714   \n",
       "\n",
       "           7         8         9  ...        11        12        13        14  \\\n",
       "14  1.555770  1.116734  0.146883  ... -2.132336  0.739735  0.452615  0.299427   \n",
       "27 -0.299485  0.821154  0.018271  ... -1.634293 -0.618584  1.354337 -1.136238   \n",
       "79  2.161531  0.390874 -0.900138  ... -0.904111  2.640541 -2.483898 -1.619484   \n",
       "\n",
       "          15        16        17        18        19  label  \n",
       "14  0.683967 -0.089078  0.609046 -0.895865 -0.578405      1  \n",
       "27  1.248243 -0.593805  0.266741  1.180665  1.212383      1  \n",
       "79 -3.676358  0.704040 -3.192003  1.669527  0.782062      1  \n",
       "\n",
       "[3 rows x 21 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_fetch_run.artifact(\"train-dataset\").as_df().sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "619375eb",
   "metadata": {},
   "source": [
    "### Training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5d2ac325",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f089febf8d0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.set_function(\n",
    "    func=\"./src/train.py\",\n",
    "    name=\"train\",\n",
    "    handler=\"train\",\n",
    "    kind=\"job\",\n",
    "    image=\"mlrun/mlrun\",\n",
    "    with_repo=True,\n",
    "    tag=\"v4\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b1a3ac58",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:05:04,113 [info] Storing function: {'name': 'train-train', 'uid': '423d664e6e684b1fb9acc9e62189d5b4', 'db': 'http://mlrun-api:8080'}\n",
      "> 2023-05-17 09:05:04,362 [info] Job is running in the background, pod: train-train-7z8z8\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>new-ci-cd-proj-shapira</td>\n",
       "      <td><div title=\"423d664e6e684b1fb9acc9e62189d5b4\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/423d664e6e684b1fb9acc9e62189d5b4/overview\" target=\"_blank\" >...2189d5b4</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>May 17 09:05:08</td>\n",
       "      <td>completed</td>\n",
       "      <td>train-train</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=shapira</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=shapira</div><div class=\"dictlist\">mlrun/client_version=1.3.1-rc5</div><div class=\"dictlist\">mlrun/client_python_version=3.7.6</div><div class=\"dictlist\">host=train-train-7z8z8</div><div class=\"dictlist\">release=v3</div></td>\n",
       "      <td><div title=\"store://artifacts/new-ci-cd-proj-shapira/data-fetch-data-fetch_train-dataset:860b9700cd3f4724a0669cb7c9732beb\">train_data</div><div title=\"store://artifacts/new-ci-cd-proj-shapira/data-fetch-data-fetch_test-dataset:860b9700cd3f4724a0669cb7c9732beb\">test_data</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">accuracy=0.85</div><div class=\"dictlist\">f1_score=0.8421052631578948</div><div class=\"dictlist\">precision_score=1.0</div><div class=\"dictlist\">recall_score=0.7272727272727273</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result059a5254\" title=\"files/v3io/projects/new-ci-cd-proj-shapira/artifacts/train-train/0/feature-importance.html\">feature-importance</div><div title=\"v3io:///projects/new-ci-cd-proj-shapira/artifacts/train-train/0/test_set.parquet\">test_set</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result059a5254\" title=\"files/v3io/projects/new-ci-cd-proj-shapira/artifacts/train-train/0/confusion-matrix.html\">confusion-matrix</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result059a5254\" title=\"files/v3io/projects/new-ci-cd-proj-shapira/artifacts/train-train/0/roc-curves.html\">roc-curves</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result059a5254\" title=\"files/v3io/projects/new-ci-cd-proj-shapira/artifacts/train-train/0/calibration-curve.html\">calibration-curve</div><div title=\"v3io:///projects/new-ci-cd-proj-shapira/artifacts/train-train/0/model/\">model</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result059a5254-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result059a5254-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result059a5254\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result059a5254-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/423d664e6e684b1fb9acc9e62189d5b4/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:05:25,610 [info] run executed, status=completed: {'name': 'train-train'}\n"
     ]
    }
   ],
   "source": [
    "train_run = project.run_function(\n",
    "    function=\"train\",\n",
    "    inputs={\n",
    "        \"train_data\": data_fetch_run.outputs[\"train-dataset\"],\n",
    "        \"test_data\": data_fetch_run.outputs[\"test-dataset\"],\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74feafe7",
   "metadata": {},
   "source": [
    "### Serving the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a04b55d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: mlrun&#45;flow Pages: 1 -->\n",
       "<svg width=\"291pt\" height=\"52pt\"\n",
       " viewBox=\"0.00 0.00 290.59 52.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 48)\">\n",
       "<title>mlrun&#45;flow</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-48 286.5921,-48 286.5921,4 -4,4\"/>\n",
       "<!-- _start -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>_start</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"38.5476,-4.0493 40.698,-4.1479 42.8263,-4.2953 44.9236,-4.4913 46.9815,-4.7353 48.9917,-5.0266 50.9463,-5.3645 52.8377,-5.7479 54.6587,-6.1759 56.4025,-6.6472 58.0628,-7.1606 59.634,-7.7147 61.1107,-8.308 62.4882,-8.9388 63.7625,-9.6054 64.9302,-10.3059 65.9882,-11.0385 66.9343,-11.8012 67.7669,-12.5918 68.4849,-13.4082 69.0878,-14.2481 69.5758,-15.1093 69.9496,-15.9894 70.2102,-16.886 70.3595,-17.7965 70.3997,-18.7186 70.3334,-19.6497 70.1636,-20.5873 69.8937,-21.5287 69.5276,-22.4713 69.0691,-23.4127 68.5225,-24.3503 67.8923,-25.2814 67.1831,-26.2035 66.3996,-27.114 65.5464,-28.0106 64.6285,-28.8907 63.6504,-29.7519 62.617,-30.5918 61.5329,-31.4082 60.4024,-32.1988 59.2299,-32.9615 58.0197,-33.6941 56.7755,-34.3946 55.5012,-35.0612 54.2002,-35.692 52.8757,-36.2853 51.5309,-36.8394 50.1684,-37.3528 48.7908,-37.8241 47.4003,-38.2521 45.9989,-38.6355 44.5886,-38.9734 43.1708,-39.2647 41.7472,-39.5087 40.3189,-39.7047 38.8872,-39.8521 37.4531,-39.9507 36.0175,-40 34.5815,-40 33.146,-39.9507 31.7119,-39.8521 30.2801,-39.7047 28.8519,-39.5087 27.4282,-39.2647 26.0105,-38.9734 24.6001,-38.6355 23.1988,-38.2521 21.8083,-37.8241 20.4306,-37.3528 19.0681,-36.8394 17.7233,-36.2853 16.3989,-35.692 15.0979,-35.0612 13.8236,-34.3946 12.5794,-33.6941 11.3691,-32.9615 10.1967,-32.1988 9.0662,-31.4082 7.982,-30.5918 6.9486,-29.7519 5.9706,-28.8907 5.0526,-28.0106 4.1995,-27.114 3.4159,-26.2035 2.7067,-25.2814 2.0765,-24.3503 1.53,-23.4127 1.0715,-22.4713 .7053,-21.5287 .4355,-20.5873 .2657,-19.6497 .1993,-18.7186 .2395,-17.7965 .3888,-16.886 .6495,-15.9894 1.0232,-15.1093 1.5112,-14.2481 2.1141,-13.4082 2.8321,-12.5918 3.6647,-11.8012 4.6109,-11.0385 5.6689,-10.3059 6.8365,-9.6054 8.1108,-8.9388 9.4884,-8.308 10.9651,-7.7147 12.5362,-7.1606 14.1966,-6.6472 15.9404,-6.1759 17.7614,-5.7479 19.6528,-5.3645 21.6074,-5.0266 23.6176,-4.7353 25.6755,-4.4913 27.7728,-4.2953 29.901,-4.1479 32.0515,-4.0493 34.2154,-4 36.3837,-4 38.5476,-4.0493\"/>\n",
       "<text text-anchor=\"middle\" x=\"35.2995\" y=\"-18.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">start</text>\n",
       "</g>\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title></title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"164.5991,-14.5442 164.5991,-29.4558 148.7828,-40 126.4153,-40 110.5991,-29.4558 110.5991,-14.5442 126.4153,-4 148.7828,-4 164.5991,-14.5442\"/>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"168.5991,-12.4034 168.5991,-31.5966 149.9939,-44 125.2042,-44 106.5991,-31.5966 106.5991,-12.4034 125.2042,0 149.9939,0 168.5991,-12.4034\"/>\n",
       "</g>\n",
       "<!-- _start&#45;&gt; -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_start&#45;&gt;</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M69.9335,-22C78.4325,-22 87.6131,-22 96.3878,-22\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"96.437,-25.5001 106.437,-22 96.4369,-18.5001 96.437,-25.5001\"/>\n",
       "</g>\n",
       "<!-- model -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>model</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"243.5956\" cy=\"-22\" rx=\"38.9931\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"243.5956\" y=\"-18.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">model</text>\n",
       "</g>\n",
       "<!-- &#45;&gt;model -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>&#45;&gt;model</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M168.8282,-22C176.8166,-22 185.6035,-22 194.2596,-22\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"194.3053,-25.5001 204.3053,-22 194.3052,-18.5001 194.3053,-25.5001\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f089fe9fb50>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a serving function object\n",
    "serving = mlrun.new_function(\n",
    "    name=\"serving\", kind=\"serving\", image=\"mlrun/mlrun\", tag=\"v4\"\n",
    ")\n",
    "\n",
    "# Add a model to the model serving function object\n",
    "serving.add_model(\n",
    "    key=\"model\",\n",
    "    model_path=train_run.outputs[\"model\"],\n",
    "    class_name=\"mlrun.frameworks.sklearn.SklearnModelServer\",\n",
    ")\n",
    "\n",
    "# Plot the serving graph\n",
    "serving.spec.graph.plot(rankdir=\"LR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ddafe3c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:05:25,815 [info] function spec saved to path: ././clone/function_spec/serving.yaml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.serving.ServingRuntime at 0x7f08a0056150>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save the function spec into a .yaml file and register it in the project\n",
    "serving.export(target=f\"./{project.context}/function_spec/serving.yaml\")\n",
    "project.set_function(func=\"./function_spec/serving.yaml\", name=\"serving\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c2675db",
   "metadata": {},
   "source": [
    "```{admonition} CI/CD configuration considerations\n",
    "- When creating a serving function, the function spec contains metadata of the function steps or the serving function \n",
    "models. Therefore, you need to create a function.yaml file by using the {py:class}`~mlrun.runtimes.BaseRuntime.export` \n",
    "method that exports the function object to a yaml file (that is saved in the function_spec folder). Then set the function \n",
    "with this yaml file. This approach saves all of the function spec for future deployments. (If you don't set the function yaml, you'll need to set the function steps or models to the function when loading the project.)\n",
    "For example:<br>\n",
    "       \n",
    "       <function object>.export('./function_spec/model_training.yaml')\n",
    "       project.set_function(\n",
    "             func=\"training.yaml\",name='training',with_repo=True,kind='serving')\n",
    "       \n",
    "- Additionally, if you want to change the default function spec values, e.g. resources, node-selector and more, and want to \n",
    "make this change constant, you need to create a yaml function file and use the yaml function in the `set_function` method.\n",
    "- When setting a nuclio function, the function handler is a combination of the `file_name::function_handler`, for example:\n",
    "      ```\n",
    "      project.set_function(name='nuclio',handler='multi:multi_3',kind='nuclio',image='mlrun/mlrun',with_repo=True)\n",
    "      ```\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b740ee1",
   "metadata": {},
   "source": [
    "To deploy a remote function, e.g. serving and nuclio kinds, use the {py:class}`~api/mlrun.projects.deploy_function` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "93d0b01e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:05:25,839 [info] Starting remote function deploy\n",
      "2023-05-17 09:05:26  (info) Deploying function\n",
      "2023-05-17 09:05:26  (info) Building\n",
      "2023-05-17 09:05:26  (info) Staging files and preparing base images\n",
      "2023-05-17 09:05:26  (info) Building processor image\n",
      "2023-05-17 09:06:26  (info) Build complete\n",
      "2023-05-17 09:06:34  (info) Function deploy complete\n",
      "> 2023-05-17 09:06:37,264 [info] successfully deployed function: {'internal_invocation_urls': ['nuclio-new-ci-cd-proj-shapira-serving-v3.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['new-ci-cd-proj-shapira-serving-v3-new-ci-cd-proj-shapira.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/']}\n"
     ]
    }
   ],
   "source": [
    "# Serve the function\n",
    "serving_func = project.deploy_function(\n",
    "    function=\"serving\",\n",
    "    models=[\n",
    "        {\n",
    "            \"key\": \"model\",\n",
    "            \"model_path\": train_run.outputs[\"model\"],\n",
    "            \"class_name\": \"mlrun.frameworks.sklearn.SklearnModelServer\",\n",
    "        }\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c2192d8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_data = \"\"\"{\"inputs\":[[-0.60150011,  0.51150308,  0.25701239, -1.51777297, -1.82961288,\n",
    "         0.22983693, -0.40761625,  0.82325082,  1.1779216 ,  1.08424275,\n",
    "        -0.7031145 , -0.40608979, -0.36305977,  1.28075006,  0.94445967,\n",
    "         1.19105828,  1.93498414,  0.69911167,  0.50759757,  0.91565635]]}\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a391939c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:06:37,342 [info] invoking function: {'method': 'POST', 'path': 'http://nuclio-new-ci-cd-proj-shapira-serving-v3.default-tenant.svc.cluster.local:8080/'}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'id': '8ca4f4ef-7765-4d50-8a43-1e77a15e433f',\n",
       " 'model_name': 'model',\n",
       " 'outputs': [1]}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "serving_func.function.invoke(\"/\", my_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "215c539d",
   "metadata": {},
   "source": [
    "## Setting a workflow\n",
    "\n",
    "After you completed developing your functions (in this example: `data_fetch`, `training`, and `serving`), \n",
    "continue with creating a workflow that runs those functions serially. For more information about workflows and an \n",
    "example of a `workflow.py` file, see {ref}`build-run-workflows-pipelines`.\n",
    "\n",
    "To set a workflow to a project, use the {py:class}`~mlrun.projects.MlrunProject.set_workflow` method. This method adds or \n",
    "updates a workflow, and specifies a name and the code path in the project.yaml file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682bef8f",
   "metadata": {},
   "source": [
    "This example adds a workflow named main that points to a file located in <br>`./< project-context >/src/workflow.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0aa3c6a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "project.set_workflow(\"main\", \"./src/workflow.py\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e49ce34",
   "metadata": {},
   "source": [
    "## Running a workflow using a Git source\n",
    "To run the workflow, use the {py:class}`~mlrun.projects.MlrunProject.run` method. With {py:class}`~mlrun.projects.MlrunProject.run` you can run a workflow \n",
    "or schedule a workflow using kubeflow pipelines by specifying the workflow name or the workflow file path.\n",
    "\n",
    "To specify running remote, use `remote:local` or `remote:kfp`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6fd0aba7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>Pipeline running (id=626a345a-b67f-4eb0-9a3b-4850185ada10), <a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor-workflows/workflow/626a345a-b67f-4eb0-9a3b-4850185ada10\" target=\"_blank\"><b>click here</b></a> to view the details in MLRun UI</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: kfp Pages: 1 -->\n",
       "<svg width=\"130pt\" height=\"188pt\"\n",
       " viewBox=\"0.00 0.00 130.00 188.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 184)\">\n",
       "<title>kfp</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-184 126,-184 126,4 -4,4\"/>\n",
       "<!-- ci&#45;cd&#45;tutorial&#45;r6bsx&#45;1763204580 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>ci&#45;cd&#45;tutorial&#45;r6bsx&#45;1763204580</title>\n",
       "<polygon fill=\"#00ff00\" stroke=\"#000000\" points=\"122,-36 4,-36 0,-32 0,0 118,0 122,-4 122,-36\"/>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"118,-32 0,-32 \"/>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"118,-32 118,0 \"/>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"118,-32 122,-36 \"/>\n",
       "<text text-anchor=\"middle\" x=\"61\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">deploy&#45;serving</text>\n",
       "</g>\n",
       "<!-- ci&#45;cd&#45;tutorial&#45;r6bsx&#45;3399705660 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>ci&#45;cd&#45;tutorial&#45;r6bsx&#45;3399705660</title>\n",
       "<ellipse fill=\"#00ff00\" stroke=\"#000000\" cx=\"61\" cy=\"-90\" rx=\"33.2948\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"61\" y=\"-86.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">train</text>\n",
       "</g>\n",
       "<!-- ci&#45;cd&#45;tutorial&#45;r6bsx&#45;3399705660&#45;&gt;ci&#45;cd&#45;tutorial&#45;r6bsx&#45;1763204580 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>ci&#45;cd&#45;tutorial&#45;r6bsx&#45;3399705660&#45;&gt;ci&#45;cd&#45;tutorial&#45;r6bsx&#45;1763204580</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M61,-71.8314C61,-64.131 61,-54.9743 61,-46.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"64.5001,-46.4132 61,-36.4133 57.5001,-46.4133 64.5001,-46.4132\"/>\n",
       "</g>\n",
       "<!-- ci&#45;cd&#45;tutorial&#45;r6bsx&#45;686534511 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>ci&#45;cd&#45;tutorial&#45;r6bsx&#45;686534511</title>\n",
       "<ellipse fill=\"#00ff00\" stroke=\"#000000\" cx=\"61\" cy=\"-162\" rx=\"57.6901\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"61\" y=\"-158.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">data&#45;fetch</text>\n",
       "</g>\n",
       "<!-- ci&#45;cd&#45;tutorial&#45;r6bsx&#45;686534511&#45;&gt;ci&#45;cd&#45;tutorial&#45;r6bsx&#45;3399705660 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>ci&#45;cd&#45;tutorial&#45;r6bsx&#45;686534511&#45;&gt;ci&#45;cd&#45;tutorial&#45;r6bsx&#45;3399705660</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M61,-143.8314C61,-136.131 61,-126.9743 61,-118.4166\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"64.5001,-118.4132 61,-108.4133 57.5001,-118.4133 64.5001,-118.4132\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f08d678ae10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h2>Run Results</h2><h3>[info] Workflow 626a345a-b67f-4eb0-9a3b-4850185ada10 finished, state=Succeeded</h3><br>click the hyper links below to see detailed results<br><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>uid</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div title=\"d75565707a53422aae5680a4d1cc13bd\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/d75565707a53422aae5680a4d1cc13bd/overview\" target=\"_blank\" >...d1cc13bd</a></div></td>\n",
       "      <td>May 17 09:07:23</td>\n",
       "      <td>completed</td>\n",
       "      <td>train</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">accuracy=1.0</div><div class=\"dictlist\">f1_score=1.0</div><div class=\"dictlist\">precision_score=1.0</div><div class=\"dictlist\">recall_score=1.0</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"56900a2cfcc4460eba5fc8572476131c\"><a href=\"https://dashboard.default-tenant.app.cust-cs-il-3-5-2.iguazio-cd2.com/mlprojects/new-ci-cd-proj-shapira/jobs/monitor/56900a2cfcc4460eba5fc8572476131c/overview\" target=\"_blank\" >...2476131c</a></div></td>\n",
       "      <td>May 17 09:06:53</td>\n",
       "      <td>completed</td>\n",
       "      <td>data-fetch</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "626a345a-b67f-4eb0-9a3b-4850185ada10"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run the workflow named main and wait for pipeline completion (watch=True)\n",
    "project.run(\"main\", watch=True, engine=\"remote:kfp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50678486",
   "metadata": {},
   "source": [
    "### Running a scheduled workflow\n",
    "\n",
    "For more information about scheduling workflows, see {ref}`scheduled-jobs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d4fc3f98",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2023-05-17 09:09:34,482 [warning] WARNING!, you seem to have uncommitted git changes, use .push()\n",
      "> 2023-05-17 09:09:34,485 [info] executing workflow scheduling 'workflow-runner-main' remotely with kfp engine\n",
      "> 2023-05-17 09:09:34,489 [info] Storing function: {'name': 'main', 'uid': '88a2eecd5cd14c339529f2c7ced3a011', 'db': 'http://mlrun-api:8080'}\n",
      "> 2023-05-17 09:09:34,854 [info] task schedule modified: {'schedule': '0 * * * *', 'project': 'new-ci-cd-proj-shapira', 'name': 'main'}\n"
     ]
    }
   ],
   "source": [
    "project.run(\"main\", schedule=\"0 * * * *\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d65de8ea",
   "metadata": {},
   "source": [
    "## Setting and registering the project artifacts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa57e2f4",
   "metadata": {},
   "source": [
    "To register artifacts to a project, use the {py:class}`~mlrun.projects.MlrunProject.set_artifact` method. By adding/setting an artifact in the project spec, they are registered upon loading the project. \n",
    "In general, use this method when you want to register an artifact when loading a project, for example:\n",
    "* You developed a model artifact in the development system and you want to use this model file in production.\n",
    "* There are artifacts you want to register by default when you load or create a project.\n",
    "\n",
    "```{admonition} Registering artifacts in multiple environments\n",
    "To register artifacts in multiple environments, for example dev and production, you must upload your artifacts to a remote storage e.g. S3. You can change your project artifact path using the MLRun UI or MLRun, for example:\n",
    "```\n",
    "```\n",
    "project.artifact_path='s3:<bucket-name/..'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0c5ea956",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the model object to register\n",
    "model_obj = project.get_artifact(\"model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "230e7922",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "v3io:///projects/new-ci-cd-proj-shapira/artifacts/626a345a-b67f-4eb0-9a3b-4850185ada10/train/0/model/\n"
     ]
    }
   ],
   "source": [
    "# print a target path\n",
    "print(model_obj.target_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7c6d5b01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'model.pkl'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print model file\n",
    "model_obj.model_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ab0a4d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "# register the model artifact to the project\n",
    "project.set_artifact(\n",
    "    key=\"model-test\",\n",
    "    artifact=mlrun.artifacts.ModelArtifact(model_file=model_obj.model_file),\n",
    "    target_path=model_obj.target_path,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc3fdc95",
   "metadata": {},
   "source": [
    "```{admonition} Artifact types\n",
    "By default, the artifact type is equal to `mlrun.artifacts.Artifact()`. To specify different types, use the relevant \n",
    "artifact object. Then you can specify specific parameters to the artifact object type. See more details in {ref}`artifacts`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aedeb29e",
   "metadata": {},
   "source": [
    "### Registering the runtime values together with their artifacts\n",
    "When MLRun creates an artifact there are values that are processed in runtime e.g. dataset preview or model metrics. These \n",
    "values are stored in the artifact spec. If you want to store the artifact spec for registering the artifact with those \n",
    "values, export the artifact object to a folder named `./artifacts` and set the artifact using the artifact.yaml file. For example:\n",
    "\n",
    "```\n",
    "model_obj = project.get_artifact('model')\n",
    "model_obj.export(./artifact/model_artifact.yaml)\n",
    "project.set_artifact(key='model',artifact='./model_artifact.yaml')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6370ec41",
   "metadata": {},
   "source": [
    "## Creating and saving the project YAML"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bc967f7",
   "metadata": {},
   "source": [
    "The project YAML contains metadata about the project, for example, all the functions set to the project, the artifacts, and the workflow. When you load the project, it loads with all of these functions, artifacts, and workflow.\n",
    "\n",
    "In general, MLRun uses this metadata to create objects, for example, function objects, and then uses those objects to \n",
    "run the functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "dfabc585",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kind: project\n",
      "metadata:\n",
      "  name: new-ci-cd-proj-shapira\n",
      "  created: '2023-05-17T08:51:02.127000'\n",
      "spec:\n",
      "  functions:\n",
      "  - url: ./src/data_fetch.py\n",
      "    name: data-fetch\n",
      "    kind: job\n",
      "    image: mlrun/mlrun\n",
      "    handler: data_fetch\n",
      "    with_repo: true\n",
      "    tag: v3\n",
      "  - url: ./src/train.py\n",
      "    name: train\n",
      "    kind: job\n",
      "    image: mlrun/mlrun\n",
      "    handler: train\n",
      "    with_repo: true\n",
      "    tag: v3\n",
      "  - url: ./function_spec/serving.yaml\n",
      "    name: serving\n",
      "  workflows:\n",
      "  - path: ./src/workflow.py\n",
      "    name: main\n",
      "  artifacts:\n",
      "  - kind: model\n",
      "    metadata:\n",
      "      project: new-ci-cd-proj-shapira\n",
      "      key: model-test\n",
      "    spec:\n",
      "      target_path: v3io:///projects/new-ci-cd-proj-shapira/artifacts/626a345a-b67f-4eb0-9a3b-4850185ada10/train/0/model/\n",
      "      model_file: model.pkl\n",
      "    status:\n",
      "      state: created\n",
      "  conda: ''\n",
      "  source: git://github.com/GiladShapira94/example-ci-cd.git#master\n",
      "  origin_url: git://github.com/GiladShapira94/example-ci-cd.git#refs/heads/master\n",
      "  load_source_on_run: true\n",
      "  desired_state: online\n",
      "  owner: shapira\n",
      "status:\n",
      "  state: online\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(project.to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "576251a2",
   "metadata": {},
   "source": [
    "To export the project content to the yaml file (saved in the project context) and save the project in the database, use the {py:class}`~mlrun.projects.MlrunProject.save` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "37313a2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.projects.project.MlrunProject at 0x7f08cf86d3d0>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Export the yaml file and save the project\n",
    "project.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "627c8ce6",
   "metadata": {},
   "source": [
    "## Creating and pushing changes to your Git repo or archive file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ed284b8",
   "metadata": {},
   "source": [
    "### Creating a Git remote\n",
    "\n",
    "If you do not clone any files and you do not have any git remotes configured in your local folder you can use {py:class}`~mlrun.projects.MlrunProject.create_remote`. This method creates a git remote and adds the remote to the project as the project source.\n",
    "\n",
    "For example:\n",
    "```\n",
    "project.create_remote(url='https://github.com/mlrun/example-ci-cd.git',name='mlrun-remote',branch='master')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a765e83f",
   "metadata": {},
   "source": [
    "### Pushing changes to the Git repo\n",
    "\n",
    "After you made changes in your code, push your project context to GitHub repo using {py:class}`~mlrun.projects.MlrunProject.push`.\n",
    "```\n",
    "project.push(branch='master',message='update',add=['project.yaml','./src/data_fetch.py','./src/serving.yaml','./src/train.py','./src/workflow.py'])\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
